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Abstract. The Palomar Transient Factory (PTF) is a synoptic sky survey in oper¬ 
ation since 2009. PTF utilizes a 7.1 square degree camera on the Palomar 48-inch 
Schmidt telescope to survey the sky primarily at a single wavelength (R-band) at a rate 
of 1000-3000 square degrees a night. The data are used to detect and study transient and 
moving objects such as gamma ray bursts, supernovae and asteroids, as well as variable 
phenomena such as quasars and Galactic stars. The data processing system at IPAC 
handles realtime processing and detection of transients, solar system object process¬ 
ing, high photometric precision processing and light curve generation, and long-term 
archiving and curation. This was developed under an extremely limited budget profile 
in an unusually agile development environment. Here we discuss the mechanics of this 
system and our overall development approach. 

Although a significant scientific installation in of itself, PTF also serves as the 
prototype for our next generation project, the Zwicky Transient Facility (ZTF). Begin¬ 
ning operations in 2017, ZTF will feature a 50 square degree camera which will enable 
scanning of the entire northern visible sky every night. ZTF in turn will serve as a 
stepping stone to the Large Synoptic Survey Telescope (LSST), a major NSF facility 
scheduled to begin operations in the early 2020s. 


1. The Palomar Transient Factory 

The Palomar Transient Faetory evolved from ideas germinating in the survey work 
of the 1990s. This included numerous large areal ground-based optical surveys (e.g. 
the SDSS, CFHT-LS, etc), several of which contained long-timescale synoptic com¬ 
ponents. Other experiments centered specifically on supernova detection, such as the 
High-z SNe Search and the Supernova Cosmology Project. At the same time, a push 
developed for asteroid threat detection including a mandate from the US government. 
Several projects were developed specifically focussing on lime-variable, Iransienl sci¬ 
ence including LINEAR, Palomar Quesl, fhe CRTS, and PAN-STARRS. Inleresl devel¬ 
oped al Callech for a survey lhal was slruclured specifically around various cadences 
aimed al SNe on scales larger lhan had previously been allempled. Advancemenls in 
compuler lechnology allowed for fasler processing, including lurnaround wilhin fhe 
same nighf. PTF was jumpslarled wilh inilial seed money from several sources. The 
original PTF parlners were Callech, LBNL, IPAC, U.C. Berkeley, LCO-GT, Oxford, 
Columbia, and fhe Weizmann Inslilule of Israel. Allhough many of Ihese are no longer 
members of Ihe currenl collaboralion, Ihey should be recognized here. In particular, 
critical conlribulions lo Ihe early success of PTF were made by U.C. Berkeley (Ihe 
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machine learning layer) and LBNL/NERSC (the initial image subtraction and transient 
detection pipeline). 

In 2013 the original PTF partnership expired and a new collaboration was formed. 
Known as the “intermediate” Palomar Transient Factory (or iPTF), this is intended to 
bridge the operational gap between PTF and the start of our next generation project. The 
new partners are Caltech, IPAC, the IPMU Kavli Insitute of Tokyo, Fos Alamos Na¬ 
tional Faboratory, the Oskar Klein Centre of Stockholm, the University of Wisconsin- 
Madison, the University System of Taiwan, and the Weizmann Institute. iPTF differs 
from PTF primarily in the mode of science operations. Whereas PTF concentrated on 
several interlocking science programs with different cadences that were performed ev¬ 
ery night for several years, iPTF concentrates on specific focussed science campaigns, 
rofafed on a quarferly basis. These campaigns allow observing cadences and fargefs 
fhaf mighf nof lend fhemselves well fo timesharing wifh ofher programs. 

The Infrared Processing and Analysis Cenfer (IPAC) is a mulfi-mission science 
center located on fhe campus of fhe California Insfifufe of Technology. While originally 
fhe NASA IRAS science center, if has evolved fo handle a variefy of space-based (e.g. 
ISO, Spifzer, Herschel, Planck, WISF) as well as ground-based projecfs (e.g. 2MASS, 
FSST, PTI, KOA, FCO-GT). IPAC also hosfs several major archives, including fhe 
Infrared Science Archive (IRSA), fhe NASA Fxfragalacfic Dafabase (NFD), and fhe 
NASA Exoplanef Dafabase. IPAC has long hisfory of experience wifh developmenf of 
dafa processing pipelines and archive inlerfaces. Combined wifh ifs FSST involvemenf 
(science user inlerface), IPAC was a clear til wifh PTF in ferms of science and soffware 
developmenf inferesf. 

More information abouf PTF, iPTF, and ZTF can be found af http: //ptf. caltech. 
edu 


1.1. Data Acquisition System 

A substantial cost savings was achieved by retasking an existing camera (the CFHT 


12k Mosaic Camera, Starr et al. (20001) rather than building an entirely new one. The 
camera was substantially reworked ( |Faw et al.|2009 1, the most important alteration be¬ 
ing replacement of the liquid nitrogen dewar with a mechanical cryo-cooler. One of the 
12 CCDs was inoperable, but the cost risk for repair was too great. While cosmetically 
undesirable, in terms of the primary project science goal (detecting transients) this had 
less impact than might be expected, and the survey mapping strategy results in cover¬ 
age over these regions. The camera also received a new field corrector, as well as a 
dry gas feed to prevent condensation from forming on the dewar window. The camera 
was fitted with a sliding filter exchanger capable of holding two filters at any one time 
(nominally these are g and R-band, with an Ha filter for use during the full moon). 
The vast majority of the survey is executed at R-band. This use of a single filter for 
the majority of the survey was deliberate. Since the 48-inch PTF telescope is intended 
primarily as a discovery engine, the use of a single filter maximizes the swept discovery 
volume (and hence cadence). A sliding shutter was also installed in front of the camera. 

The telescope is the 48-inch (1.2m) Oschin-Schmidt telescope at Palomar. By us¬ 
ing a pre-existing Caltech-owned facility, the cost savings was considerable over other 
projects that have constructed new telescopes. This is the same telescope used for 
the first “big data” project in astronomy, the original Palomar Observatory Sky Survey 
(POSS), as well as the later DPOSS and Quest projects. The PTF camera is mounted 
inside the telescope tube in the location of the original photographic plate holder. 
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PTF Depth of Coverage (Epochs) 
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Figure 1. Number of independent observations at R-band per location on the sky 
over the current 6-year lifespan of the PTF/iPTF project. This is an aitoff projection 
of the sky in galactic coordinates (i.e. the galactic plane extends horizontally across 
the image). Coordinate lines are celestial Right Ascension and Declination. 


Telescope operations are fully robotic, saving the time expense and time ineffi¬ 
ciencies associated with human observers. A human telescope operator at the nearby 
Hale 200-inch makes safety decisions about opening and closing the 48-inch dome. 
The robotic telescope controller autonomously follows a basic set of observation pri¬ 
orities, dynamically adapting observations to maintain the desired cadence. PTF does 
not follow a simple raster pattern on the sky. Rather, the observation pattern is science 
driven, with certain types of targets having different cadences. 

At large fraction of PTF science involves rapid transient response. New transient 
sources (SNe, asteroids) are discovered and scheduled for follow-up, sometimes in the 
same night, at the Palomar 60-inch, the 200-inch, LCO-GT, or other facilities. This 
integrated usage of the Palomar telescopes (discovery with the wide-held 48-inch with 
coordinated followup of bright sources with the 60-inch and faint sources with the 200- 
inch) actually mirrors the original Palomar design ( [MinkowsE 19721, only replacing 
the previous instruments with modem digital devices. 


1.2. Data Processing Layer (Operations) Physical Infrastructure 

Raw data is transmitted from Palomar via a microwave link, relaying to IPAC through 
the San Diego Supercomputing Center. Although currently adequate, the maximum 
bandwidth of this link will be one of the rate-limiting factors for our next generation 
project (described below). This data transmission issue is common to many synoptic 
surveys, and remains an outstanding infrastructural issue. Data processing cannot occur 
on the mountaintop. All of the images in a synoptic survey are of value, and as a result 
the data “reduction” process actually increases the overall data volume by factors of 
several. In addition, the physical plant (space, power, cooling, etc) available at Palomar 
are inadequate to host the required computing resources on-site. 

There are twenty-four compute drones, half of which are dual-cpu 8-core Sun 
systems, while the other half are dual-cpu 12-core Dells. All run Red Hat Enterprise 
linux. The fileserver and database servers are Sun systems running Solaris. AlOG 
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ethernet network is used. All of the drones were purchased new, although in some 
cases second-hand units were used for beta testing. 

Disk storage is based on Nexsan ATABeast network attached storage. In addition, 
the individual compute drones are equipped with internal scratch disks. In most cases 
these were purchased new, although two of the units consisted of used chassis populated 
with new disks. The filesystem is ZFS. In-line filesystem data compression is used 
which generally recovers about 30% of the physical storage. An unusual feature of this 
system is that the long-term storage is dual-homed and connected to both the operations 
system and the archive system. This was a financial consideration: the spinning disk is 
too expensive to replicate both an operations and archive copy. This adds an additional 
operational complexity cost, since it requires careful control of file ownership between 
the operations and archive layers. 

The PTF system lies behind the IPAC facility two-factor authentication firewall. 
The only exposed components are the transfer computer used to receive data, and the 
archive interface computers. Because PTF shares facilities infrastructure with all of 
IPAC (including a state of the art data room with support personnel), there is an addi¬ 
tional savings through economy of scale. 


1.3. Data Processing Layer (Operations) Software 


The data processing system draws on design heritage from numerous IPAC space mis¬ 
sions. The pipeline building blocks are individual modules that carry out specific data 
reduction operations (for example, deriving and removing the CCD bias pattern). These 
modules are written in a variety of languages, the most common being C, while oth¬ 
ers are in Python, IDL, and Matlab. The choice of language was generally dictated by 
the preference and proficiency of the individual coder. The interfaces to these mod¬ 
ules are regularized through use of PERL wrappers. The pipeline makes heavy use of 
community software. These include SWARP, SCAMP, Sextractor, Daophot, and the 
astrometry.net astrometric calibration suite. Early versions of the image subtraction 
pipeline also used HOTPANTS. 

The pipeline executor is also written in PERL (Laher et al. 2014[ ). The system 
divides the major data reduction tasks into individual pipelines composed of multiple 
modules; a virtual pipeline operator handles job distribution and automated daily op¬ 
erations while a human pipeline operator chooses which additional tasks to queue. A 
postgres database is central to pipeline operations and is used to track pipeline status, 
data flow, and data quality metrics. Catalog (extracted source photometry) is not loaded 
into a relational database within the operations system as early experiments showed this 
to be too costly in terms of performance, and flat file access is used instead. 

The processing of PTE data is embarrassingly parallel. The parallel quantization is 
to process all data based on spatial location. Specifically, all PTE observations occur on 
a fixed tiling scheme on the sky referred to as a “ptffield id”. The combination of fieldid, 
CCD, and filter uniquely spatially describe all data. Processes based on ensembles of 
data are processed entirely within a drone given this spatial assortment. Individual 
CCDs are sent to individual drones. Data is processed locally using the drone internal 
scratch disks. Periodically, data that needs to be shared is written back to “sandbox” 
disks, which are a large shared disk pool connected to all the drones. After all the data 
products are produced, they are then copied into long-term archive storage. A principle 
in processing systems is “keep the data close to the cpus”. This is commonly enabled 
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through application layers that manage data. In our case this is accomplished directly 
by the pipeline design and network. 

Because the data acquisition system is an autonomous robot, we do not have fore¬ 
knowledge of the events in a given night and we only learn about the observations upon 
receipt of the data. This is in contrast to most space missions, which use a shared 
uplink/downlink database, and makes the staging of pipelines more complex. Future 
versions of this pipeline will address this issue. 



NERSC Image Subtraction and 
Transient Detection/RB Pipeline 



Figure 2. Schematic flowchart of the iPTF data flow. The encircled area is pro¬ 
cessing that occurs within IPAC. The data flow has many interconnected branches 
with reliances between the segments on data products produced by other segments. 

The primary pipeline segments are: 


• Data Ingest — The raw multi-extension FITS files are pushed into a transfer/receiving 
computer on the border of the IPAC firewall. The files are registered in the 
database and placed into the raw data archive, and an initial astrometric cali¬ 
bration is made. At this stage various checks are made for inconsistent metadata 
in the incoming data products. The files are then split apart into individual CCDs 
and directed to the various drones for processing. 


Realtime Pipeline — As soon as data arrive, a realtime pipeline is triggered. 
This is primarily an image-subtraction pipeline. The data go through an ini¬ 
tial calibration (bias subtraction, flat-fielding) using pre-existing calibration files 
from a previous night. A refined photometric calibration is derived. Software 
developed in-house provides photometric, astrometric, and beam-shape match¬ 
ing between the incoming frame and the deeper reference images, producing a 
frame where all of the static objects have been subtracted out. In-house code is 
used for beam-matching implementing the Bramich algorithm ( Bramich|2008 1. 
A source detector is run, creating transient candidates. These candidates are then 
further machine-vetted by a machine-learning algorithm called RealBogus, de- 
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veloped originally at U.C. Berkeley and now under further development by JPL 
and LANL. The resulting eandidate lists are then sent to the extragalaetie mar¬ 
shall as likely real transients. In addition, a “streak deteetor”, also developed 
in-house (originally for deteeting aireraft and satellite trails), is used to deteet 
extremely fast-moving solar system objeets. These are also vetted by a maehine- 
learning algorithm. All of these eandidates are then direeted to a moving objeet 
pipeline, whieh eonstruets potential traeklets aeross multiple observations, result¬ 
ing in lists of solar system objeets. This pipeline eurrently eompletes proeessing 
roughly 20 minutes after the data are taken. However, beeause jobs are staeked, 
this is a eonstant phase lag and the proeessing keeps up with the data flow. 

• Photometrie Pipeline — When the last observations for a given night are sent, 
they inelude an “end of night” signal indieating that no further data is expeeted. 
At this point a pipeline triggers whieh is intended to produee the highest qual¬ 
ity data produets possible, mirroring the usual data reduetion steps used by as¬ 
tronomers when hand-redueing data. Speeifieally, data from the entire night 
is used to eonstruet the best possible ealibration for the Hatfield and bias re¬ 
sponse, and to model the photometrie ealibration to eompensate for ehanging at- 
mospherie eonditions. The astrometrie ealibration is tied to a eombined UCAC4- 
SDSS eatalog, and is presented in both the PV and TAN-SIP representations. 
The photometrie ealibration is tied to the SDSS, using observations of all SDSS- 
overlapping fields throughout the night, and in-house software to propagate the 
ealibration to non-SDSS fields. Along with the redueed images, matehing bit- 
masks are ereated whieh eneode information sueh as radiation hits, CCD bleeds, 
saturation, and filter ghosts. Souree eatalogs are generated both with aperture 
photometry (tuned for galaxies) and psf-fitting (ideal for stars). The resulting 
images and eatalogs are generally flux-ealibrated to 2-3% aeeuraey, with astro¬ 
metrie aeeuraey of 0.15". 

• Referenee Image Pipeline — This pipeline generates deep images of the sky 
(ealled “referenee images”), whieh in turn are the basis for the image subtrae- 
tion stage in the realtime pipeline. The inputs are the images produeed by the 
photometrie pipeline. Various eheeks are made to ensure that only the highest 
quality data gets eoadded, and at present between 5 and 50 frames are used for 
input. A new photometrie and astrometrie ealibration is derived in whieh the 
input frames are speeifieally aligned to eaeh other, whieh is more aeeurate than 
relying on alignment through a seeondary eatalog. In-house software is used to 
derive the eoaddition weights and perform the aetual staeking. Deep referenee 
eatalogs (used by the final pipeline layer) are extraeted using both aperture pho¬ 
tometry and psf-fitting. These form the basis of the eatalog of the statie sky. As 
diseussed, these referenee images are made on a tile, ehip, and filter basis. This 
point is important - the resulting referenee images are observed with the same in¬ 
strument and filter, pointed in nearly the same loeation on the sky. This minimizes 
issues during image subtraetion with matehing aeross effeetive instruments. 

• Light Curve Pipeline — This pipeline eonneets the individual epoehal measure¬ 
ments with aetual astrophysieal objeets, and then refines the photometry to pro¬ 
duee high aeeuraey light eurves. Eaeh night, as data are eolleeted, a souree 
matehing pipeline eonneets eaeh deteetion of eaeh objeet based on spatial prox¬ 
imity to a souree loeated in the deeper referenee eatalogs. This approaeh is more 
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successful than a generalized direct cross-match between epochal catalogs. This 
is because variations in spatial resolution from the atmosphere create ambiguities 
in the cross-matching, which in a generalized catalog approach quickly develops 
an exponentially large number of potential linkages. Once the cross-matches are 
calculated, roughly once a week or month photometric corrections are generated. 
It is assumed that the magnitude zeropoints for each image are in error by small 
amounts, presumably due to uncertainties in the propagation of the SDSS de¬ 
rived magnitudes, non-photometric conditions, etc. For each tile, CCD, and filter 
combination an analysis is performed to find fhe objecfs wifhin fhe field wifh fhe 
lowesf variances. These are fhen assumed fo be sialic, and used lo anchor fhe 
calibration for each individual image. This resulls in a fable of della-correcfions 
lhal can be applied fo each poinl in a lighl curve, affer cross-referencing fhe par- 
enf image. The end resull are lighl curves wifh an accuracy of a few millimag, 
allhough Ihese are dominafed by poisson noise al magniludes fainler lhan 17. 

1.4. Archive Layer (IRSA) 

The funclion of fhe archive layer is lo provide a means of discovering dafa producfs 
produced by fhe operations layer, and fo deliver Ihose dafa producfs lo whatever Iheir 
end use mighf be. The PTF and ZTF dafa archive layers were developed by fhe Infrared 
Science Archive (IRSA) group wifhin IPAC. IRSA is a mulfimission science archive, 
providing dafa services for Spilzer, 2MASS, WISE and olhers. IRSA has developed 
common loolkils upon which Ihese mission archive interfaces were builf. Leveraging 
Ibis capabilily allowed a relafively low cosl archive fo be developed quickly. 



Figure 3. PTF image search/discovery, exploration, and download tool at the In¬ 
frared Science Archive, based on a shared toolkit also used for Spitzer and WISE. 
Common usage pattern is for users to use the tool to rapidly find what sort of data 
holdings exist within the PTF archive. 

After data is processed by the operations system at the end of each day, possession 
of the data is passed to IRSA. Image data is passed by file reference and ownership 
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change. Catalog (database) data is passed via load files generated by the operations 
system, which are then ingested into an IRSA specific dafabase. Several faclors drive 
Ibis separafion befween fhe archive and operafions layers. The archive is infended as a 
reliable long-term storage sysfem. This requires a guaranfee fhaf fhe files confinue fo 
exisf on fhe shared disk sysfem. Second, fhis allows fhe archive sysfem fo absfracf fhe 
file sforage media and pafhs, allowing fransparenf fulure hardware replacemenfs and 
upgrades. Finally, fhe underlying sforage schema required by fhe archive technology 
may currenfly be or will evolve differenlly from fhaf of fhe operations sysfem. 

The archive currenfly offers several differenl services. The firsl of fhese is a GUI 
web-based dafa search fool and visualizer. Searches are enabled by name and posifion. 
In addifion, fhere is a moving objecf based search fhaf correlafes an ephemeris againsf 
fhe posifion and fime of fhe PTF observafions. Images may be previewed fhrough a 
live display pane. Bofh images and cafalog files may be refrieved in bulk. The inlerface 
offers on-fhe-fly fhumbnailing, since mosf users are inferesfed in limifed regions around 
fheir fargefs of inferesf. A second interface is provided fhrough a sef of APIs fhaf allow 
hffp-based calls (e.g. via wgef or curl). This allows users to embed direcf calls fo fhe 
archive info fheir own soffware. Bofh fhis API and fhe web-based gui utilize fhe same 
underlying infraslruclure, and a resulf offer similar services. 

1.5. Science Tools Layer (Marshals) 

A crifical componenf of PTF for enabling rapid science resulfs are fhe so-called “sci¬ 
ence marshals”. The marshals are organized around scienlific fopics, and serve many 
differenl functions which are unique fo carrying oul science analysis. In fhis sense, fhey 
are somewhaf similar fo fhe envisioned LSST “even! brokers”. One of fhe mosf im- 
porfanf of fhese is fhe exlragalaclic marshall. If presenfs dafa from fhe realtime image 
subfracfion and fransienf detection pipeline, including fhumbnails of fhe large!, image 
differences, and images from olher surveys (e.g. SDSS) when available. Dafa fusion 
funclions include aulomaled cross-referencing fo olher calalogs, as well as hislorical 
relrieval of pholomelry from fhe region of inferesf. Mosf imporlanlly, fhere are infer- 
aclive funclions allowing scienlisls fo commenf on fhe resulfs, coordinate and upload 
follow-up observations, and additionally lag fhe dafa wilh new resulfs. A similar galac¬ 
tic marshall exisls for analysis of lighl curve dafa and includes functions such as phase¬ 
wrapping of lighf curves on demand, while a solar sysfem objecf marshall presenfs 
candidale fasl-moving objecls and shows prepared dafa for fhe Minor Planel Center. In 
mosf cases fhe marshalls incorporale fhe dafa archive as services, calling as needed dafa 
requesls for fhumbnails, elc. 

2. The Zwicky Transient Facility 

We are beginning construction of a new phase in the Palomar transient universe inves¬ 
tigation: the Zwicky Transient Facility. At the heart of this program is a new camera 
with a field of view of nearly 50 square degrees. The enabling technology for this cam¬ 
era is the development of inexpensive “wafer-scale” CCDs, e.g. detectors the full size 
of silicon wafers. The ZTF camera will utilize 16 6kx6k CCDs manufactured by e2v. 
Through the use of parallel readouts and modern CCD control electronics, these de¬ 
vices can be fully read in roughly 10 seconds. Currently, we plan to utilize a 30-second 
exposure time, with focal plane readout occurring during the telescope slew and settle. 
This will allow ZTF to cover 3n in 8 hours, or put more simply this will allow ZTF 
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to image the entire visible northern sky every night. This will lead to a variety of new 
science capabilities. We highlight here one of the more interesting: gravity wave local¬ 
ization. Although gravity waves have yet to be detected, it has long been acknowledged 
that the existing detectors (e.g. LIGO) can only localize events to large areas (hundreds 
of square degrees) on the sky. When first proposed, this was a significant criticism; 
it would be impossible to connect gravity wave sources to any other observations or 
indeed to any specific astronomical sources on the sky. With ZTF’s large areal cover¬ 
age, depth, and fast turnaround time, it will be possible to survey the entire probable 
region of the gravity wave event in order to catch any rapidly fading electromagnetic 
counterpart, a feat iPTF has already demonstrated with gamma ray burst sources. 

In addition to the new camera, ZTF will also require modest telescope modifi¬ 
cations. The most notable of these is the use of a bi-parting shutter external to the 
telescope tube. Because the camera mechanism is located inside the telescope, it is 
extremely important that it have as small a cross-section as possible, and this precludes 
the use of a focal plane shutter. Similarly, a filter exchanger is planned that will use 
a folding truss mechanism to stow the (2 of 3) filters not currently in use against the 
inner surface of the telescope. The camera itself has a remarkably small cross-section, 
being only slightly larger than the CCDs themselves. The camera itself is very thin (be¬ 
ing cooled via an external cryo-cooler) at roughly 4 inches thick. Guiding, as well as 
tip/tilt and focus measurements, are provided by four 2048x2048 CCDs, each of which 
is nearly a quarter degree by themselves. 

ZTF began receiving funding from the U.S. National Science Foundation (NSF) 
starting fiscal year 2015 under the Mid-Scale Innovations Program (MSIP). The MSIP 
award, together with contributions from private partners, is sufficient to enable full 
funding of the project through completion of three years of operations in 2020. The 
instrument is expected to see first light at the start of 2017. All data will be publicly 
released. This will start with the PTF image and catalog data in fall of 2015, followed 
by a light curve release in the following year. The first year of ZTF operations will be 
devoted to reference image building. Eventually, all catalog and image data as well as 
transient alerts will be released on an annual basis. 

ZTF is the most immediate predecessor to the Large Synoptic Survey Telescope 
(LSST), a major NSF-funded facility scheduled for first light in 2019 and science op¬ 
erations beginning around 2022. ZTF will possess a similar cadence to LSST, but the 
latter will have superior image quality, multiple simultaneous filters, and will be much 
deeper as the telescope is more than 6 times larger in radius. Nonetheless, the two sur¬ 
veys are actually highly complementary - in particular, ZTL’s prime discovery space 
will lie at magnitudes that are very bright for LSST. 


3. System Development Under Severe Constraints 

PTL has been enormously successful despite an extremely limited monetary budget. 
Most IPAC projects are NASA space missions, with substantially larger budgets and 
which follow a classic “waterfall” development approach: substantial effort devoted to 
science definition and requirements writing, followed by technical design requirements, 
and finally coding and testing with fixed delivery cycles, typically on six month scales. 
Such an approach was inapplicable to PTL for several reasons. Lirst, we lacked the 
funded manpower required for any effort beyond coding and testing. The core team 
consisted of roughly three people, with additional staff broughf in as needed. Major 
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contributions were also made by 3-5 graduate students and post-docs, as time permitted 
and the project goals aligned with their science goals. Second, we were on an acceler¬ 
ated timetable, where the software in its initial form had to be brought online in roughly 
one year, precluding a lengthy design phase. Finally, at the start of the project we were 
uncertain of the achievable data quality, or indeed what science goals could be ac¬ 
complished, thus precluding writing very detailed science requirements. PTF therefore 
proceeded under a very “agile” development cycle. 

A first critical element was that development remained science focussed. Astron¬ 
omy has become a highly competitive field, and fhe mosf common mefric of success 
is paper producfion. Therefore, we allowed fhe evolving nafure of fhe science possi¬ 
ble wifh PTF fo guide fhe direcfion of developmenf. Asfronomy shiffs focus on 3-5 
year fimescales, so foday’s science drivers are unlikely fo be tomorrow’s. A specific 
example has been fhe evolufion of PTF away from being solely a fransienf discovery 
engine. As if became obvious fhaf fhe sysfem could supporf high accuracy lighf curves, 
fhe emphasis for soflware developmenf has shiffed away from SNe (of which fhere are 
now fhousands specfroscopicahy confirmed) fo variable phenomena like Galacfic sfars. 

The developmenf program also emphasized rapid resulfs wifh a philosophy of 
esfablishing basic funclionalify and fhen addressing incremenfal improvemenfs and 
adding capabilifies as needed. Big dafa projecfs have dafa volumes fhaf are, by defi- 
nifion, loo large fo be amenable fo hand analysis. The dafa sysfem musl be capable of 
rendering fhe dafa info a form sufficienlly intelligible for furfher analysis and refine- 
menf; affempfing fo reach fhe end sfafe during developmenf is unlikely fo succeed. In 
order to speed developmenf we made exlremely abundanl use of community soflware 
(see Seclion 1.3), and funclionalify was valued over elegance. While fhe system does 
nol use fhe lalesl technology, if was broughf parliahy on-line in under one year, and 
science papers were wriflen almosl immedialely. 

Finally we used an exlremely agile developmenf process. Developmenf was by a 
very small group. The coders and leads were all scienlisls. IPAC has always insisled on 
deep involvemenl by research aslronomers in fhe developmenf process; Ihis ensures fhaf 
fhe soflware produces resulfs fhaf will acluahy meel fhe needs of fhe science users. The 
use of sludenls and posl-docs also accelerales fhe developmenf, since Ihey have a vesled 
inleresl in fhe soflware acluahy working. In fhe long term Ihis creales susfainabilily 
issues, bul in fhe shorl term resulfs in produclive soflware al low cosl and is a viable 
approach depending on fhe scope of fhe projecl. 

In conclusion, while our efforls were clearly dehned in many ways by fhe limila- 
lions of our budgel, PTF (and soon fo be ZTF) are clear examples of a high science per 
dollar projecl made possible by properly aligning soflware developmenf and science 
goals. 
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